Skip to content

[common] Fix file system leaks when the Hadoop file system cache is disabled - #8962

Open
wombatu-kun wants to merge 1 commit into
apache:masterfrom
wombatu-kun:issue/8548-hadoop-fileio-close-owned-filesystems
Open

[common] Fix file system leaks when the Hadoop file system cache is disabled#8962
wombatu-kun wants to merge 1 commit into
apache:masterfrom
wombatu-kun:issue/8548-hadoop-fileio-close-owned-filesystems

Conversation

@wombatu-kun

@wombatu-kun wombatu-kun commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #8548. With fs.<scheme>.impl.disable.cache=true every HadoopFileIO creates its own FileSystem, and nothing ever released them, so connector threads accumulate (reported with AliyunOSSFileSystem).

HadoopFileIO.close() now releases the file systems it owns, mirroring Hadoop's own branch in FileSystem.get(URI, Configuration): ours only when fs.<scheme>.impl.disable.cache is set for the path's scheme, matched as written. Otherwise the instance lives in Hadoop's global cache, is shared JVM wide, and must not be closed here.

Supporting fixes, without which that close is unreachable:

  • ResolvingFileIO and PluginFileIO did not forward close() to their delegates, unlike CachingFileIO. As a result OSSFileIO.close() had never run in production.
  • HadoopSecuredFileSystem did not override close().
  • FileIO.checkAccess discarded the probe FileIO it loads to test access, leaking one file system per FileIO.get() with no close() call involved.

close() is terminal, otherwise the wrappers silently re-created their delegates and the leak returned.

Out of scope: the per-module HadoopCompliantFileIO copies take file systems from static caches and need a separate design. Note for reviewers: RESTTokenFileIO's eviction listener already calls closeQuietly, so it now really closes file systems; the ownership gate limits that to disable.cache=true.

Tests

HadoopFileIOTest (new) plus cases in ResolvingFileIOTest, PluginFileIOTest and HadoopSecuredFileSystemTest: owned closed, cached untouched and still shared, ownership per scheme, failing close does not skip the rest, idempotency, creation race, use after close, no leaked access probe.

paimon-common 12198 tests and paimon-core rest/catalog/fs 342 tests green on JDK 8; checkstyle and spotless clean.

@wombatu-kun
wombatu-kun force-pushed the issue/8548-hadoop-fileio-close-owned-filesystems branch from 62d8e28 to 28f7bf0 Compare July 31, 2026 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] HadoopFileIO does not close per-instance FileSystems, causing executor thread growth with disabled Hadoop FS cache

1 participant